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FOREWORD 


The  Defense  Logistics  Agency  (DLA)  Vendor  Rating  Systems  was 
developed  to  provide  information  to  buyers  on  the  past 
performance  of  contractors  for  bid  evaluation  in  support  of  "best 
value"  contracting  initiatives.  The  rating  system  systematically 
collects  and  analyzes  historical  data  to  provide  consistent 
measures  of  the  vendor's  ability  to  supply  conforming  materiel, 
on  time.  This  system  was  developed  under  the  guidance  of  a  Study 
Advisory  Group  (SAG) .  The  SAG,  knowledgeable  experts  in  the 
areas  of  contracting,  production  and  quality  assurance,  provided 
valuable  contributions  to  the  development  of  the  prototype  model. 
Representing  each  Supply  Center  and  DLA  headquarters,  the  SAG 
provided  detailed  advice  and  direction  as  well  as  solutions  to 
many  problems  encountered  during  development.  We  thank  all 
members  of  this  group  for  their  cooperative  spirit  and  teamwork 
that  led  to  the  timely  completion  o£.  this  effort. 
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EXECUTIVE  SUMMARY 


Traditionally,  DLA  has  awarded  an  overwhelming  majority  of  its 
contracts  to  the  low  bidder.  In  an  effort  to  move  towards  the 
buying  practices  of  private  industry,  Total  Quality  Management 
initiatives  in  the  late  1980's  led  to  increased  efforts  to 
consider  other  factors  in  award  decisions.  All  DLA  centers  have 
implemented  a  program,  under  different  names,  that  permit  proven, 
superior  performers  to  obtain  awards  with  bids  above  low  bidder. 
These  programs  have  been  successful  but  limited  in  scope  because 
participation  by  contractors  is  restricted  due  to  the  extensive 
manual  effort  required  by  the  contractor  and  the  DLA  Supply 
Center  to  administer  the  program.  As  a  result,  the  DLA  Vendor 
Rating  System  (DVRS)  has  been  developed  to  automate,  to  the 
maximum  extent  possible,  the  data  collection  and  analysis  of 
contractor  performance,  so  that  the  information  can  be  used  in  a 
broader  range  of  applications. 

In  the  development  of  a  prototype  model  for  use  at  the  Defense 
General  Supply  Center  (DGSC) ,  two  high  level  factors  other  than 
price  are  measured.  Delivery  performance,  a  measure  of  the 
contractor's  past  ability  to  deliver  contract  lines  on  time, 
considers  the  frequency  of  delinquent  contract  lines  and  the 
duration  of  lateness.  Quality  performance,  measuring  historical 
nonconformance  rates  of  a  contractor,  is  based  on  customer 
complaints  and  lab  test  results.  Ratings  are  developed  at  the 
Federal  Supply  Class  (FSC)  level  and  are  generally  based  on  2 
years  of  past  performance.  The  model  has  been  designed  to  accept 
additional  factors,  should  management  decide  to  expand  the 
definition  of  performance. 
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To  use  DVRS,  buyers  will  access  the  ratings  through  the  DLA  Pre 
Award  Contracting  System  (DPACS) .  DVRS  is  designed  to  run  in  a 
batch  mode  (monthly)  on  the  Information  Processing 
Center-Richmond  (IPCR)  mainframe  computer.  Vendor  ratings  are 
downloaded  to  the  DPACS  system  to  the  personal  computer  of  the 
buyers.  The  ratings  will  appear  automatically  on  various  DPACS 
screens  to  facilitate  bid  evaluation  and  other  processes.  Also, 
the  ratings  and  supporting  data  are  archived  for  purposes  of 
contract  review  and  defense  of  protests. 

Preliminary  testing  of  the  DVRS  model  has  been  successfully 
completed.  Ongoing  functional  testing  will  be  done  at  DGSC,  with 
results  expected  in  early  FY  93.  The  major  obstacle  to  full 
scale  implementation  of  DVRS  appears  to  be  accuracy  of  the 
historical  data  used  by  DVRS  to  compute  ratings.  Efforts  are 
underway  at  DGSC  and  elsewhere  to  analyze  the  causes  of  incorrect 
data  and  to  prevent  entry  of  incorrect  data  into  our  systems  for 
those  few  key  data  fields  used  by  DVRS.  Manual  correction  of 
known  or  suspected  errors  is  being  performed  at  DGSC  for  those 
FSCs  being  tested. 

Implementation  of  DVRS  throughout  DLA  is  constrained  only  by  the 
need  to  process  DVRS  at  IPCR.  As  the  other  hardware  Supply 
Centers  migrate  data  processing  to  IPCR  through  May  1993,  the 
DVRS  prototype  can  be  adapted  for  wider  use.  However,  DLA-wide 
use  may  require  additional  standardization  of  post-award  and 
quality  assurance  processes  at  each  of  the  DLA  Supply  Centers. 

The  DLA  Operations  Research  and  Economic  Analysis  Management 
Support  Office  is  planning  future  projects  to  provide  analytical 
support  necessary  to  transform  this  prototype  model  to  a 
production  system. 
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SECTION  1 
INTRODUCTION 


1.1  BACKGROUND 

In  the  1980's,  under  the  influence  of  Dr.  W.  Edwards  Deming's 
quality  philosophy  and  intense  foreign  competition,  U.S. 
manufacturing  firms  have  adopted  purchasing  strategies  geared 
toward  close  buyer-supplier  relationships  with  fewer  suppliers  of 
proven  high  quality  performance.  The  Department  of  Defense, 
though  limited  by  law  through  the  Federal  Acquisition  Regulation, 
has  encouraged  its  activities  to  adopt  commercial  buying 
practices  where  possible.  The  Defense  Logistics  Agency  (DLA) , 
because  of  its  extremely  high  volume  of  relatively  low  dollar 
value  awards,  has  had  very  little  experience  in  using  factors 
other  than  price  to  award  contracts.  Bids  from  extremely  poor 
performers  are  occasionally  eliminated  through  a 
nonresponsibility  determination.  In  the  late  1980 's  and  early 
1990 's,  DLA  hardware  centers  initiated  programs  to  recognize  and 
reward  suppliers  with  outstanding  track  records.  Those  suppliers 
that  applied  for  and  could  prove  that  they  met  very  demanding 
performance  criteria  were  placed  on  an  exclusive  list  by  Federal 
Supply  Class  (FSC)  and  could  receive  awards  even  if  their  bids 
were  up  to  20  percent  higher  than  the  low  bidder. 

Both  nonresponsibility  determination  and  premium  awards  are 
limited  in  scope  and  application  because  of  the  manual  effort 
involved  by  the  contractor  and  government.  These  tools  only 
affect  the  very  few  extremely  high  and  extremely  low  performers. 
Most  competitive  awards  are  still  primarily  determined  by 
selecting  the  low  bidder  from  a  group  of  vendors  that  have 
neither  extremes  in  their  performance  history.  Nonetheless,  often 
the  difference  in  price  between  the  low  bidder  and  the  next  to 
low  bidder  may  be  small,  yet  the  past  performance  of  the  low 
bidder  may  be  significantly  worse.  The  "best  value"  to  the 
government,  considering  post  award  costs  associated  with  late 
deliveries  and  quality  resolution,  may  result  in  a  judgement  by 
the  contracting  officer  to  award  the  contract  to  other  than  the 
low  bidder. 

To  provide  the  information  needed  to  make  a  "best  value" 
decision,  DLA  Directorate  of  Contracting,  Policy  Branch  ( DLA-PPR) 
and  DLA  Directorate  of  Quality  Assurance,  Logistics  Management 
Division  (DLA-QL)  tasked  the  DLA  Operations  Research  and  Economic 
Analysis  Office  (DLA-LO)  to  develop  a  system  that  collects 
historical  performance  data,  systematically  analyzes  the  data  to 
produce  vendor  ratings  and  provides  the  ratings  and  supporting 
information  to  the  buyer  for  bid  evaluation.  This  request  was 
made  after  DLA-LO  had  completed  a  feasibility  study  in  March  1991 
that  verified  the  technical  feasibility  of  the  concept. 
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1.2 


SCOPE 


The  DLA  Vendor  Rating  System  (DVRS)  prototype  was  developed  based 
on  data  from  the  Defense  General  Supply  Center  (DGSC)  from  March 
1989  to  September  1991.  The  model  is  designed  for  use  at  all  DLA 
hardware  centers,  but  because  DVRS  uses  Standard  Automated 
Material  Management  System  (SAMMS)  files  and  DLA  Integrated  Data 
Bank  (DIDB)  files  (which  only  reside  at  Information  Processing 
Center-Richmond  (IPCR)),  the  model  cannot  be  used  for  other 
hardware  centers  until  SAMMS  processing  migrates  to  IPCR. 
Contractor  performance,  as  measured  by  DVRS,  is  limited  to  two 
main  areas  of  quality  assurance  and  delivery  effectiveness. 
Resolution  of  DVRS  measures  will  be  at  the  FSC  level  and  above. 

1.3  OBJECTIVES 

The  objectives  of  this  project  are: 

(1)  To  develop  an  automated  system  to  systematically 
measure  contractors  past  performance. 

(2)  To  demonstrate  DVRS  ratings  accurately  reflect 
contractor  performance. 

(3)  To  provide  buyers  access  to  current  ratings  at  the 
point  of  bid  evaluation. 

1.4  DOCUMENT  OVERVIEW 

The  purpose  of  this  document  is  to  provide  a  description  of  the 
DVRS  prototype  model  and  to  document  the  methodology  used  to 
develop  DVRS.  Two  additional  documents  will  be  published  for 
DVRS.  A  technical  report  by  DLA  Operations  Research  and  Economic 
Analysis  Management  Support  Office  (DLA-DORO)  will  provide  source 
code  and  Automated  Data  Processing  (ADP)  information  necessary 
for  program  maintenance.  A  user's  manual  will  be  published  by 
DLA-PPR.  This  report  will  supplement  the  user's  manual  and  serve 
as  a  baseline  for  future  model  enhancements. 
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SECTION  2 
METHODOLOGY 


2.1 


ANALYSIS  AND  MODEL  DESCRIPTION 


2.1.1  GENERAL  MODEL  DESCRIPTION 

The  overall  objective  of  the  DVRS  model  is  to  provide  the  DLA 
buying  community  with  an  effective  tool  to  evaluate  competing 
vendors'  bid  prices  against  each  vendor's  past  performance  of 
providing  quality  material  in  a  timely  manner.  This  approach 
provides  the  buyers  with  a  mechanism  for  determining  the  Best 
Value  during  the  contract  award  process.  As  the  objective 
states,  a  vendor's  performance  can  be  broken  into  two  distinct 
areas.  These  areas  are  an  assessment  of  the  quality  of  material 
provided  to  the  Agency  and  whether  or  not  the  contracted  material 
was  delivered  in  a  timely  manner.  This  section  focuses  on 
developing  the  model  methodology  for  assessing  the  vendor's 
performance. 

2.1.2  QUALITY  ASSURANCE  VENDOR  RATINGS 


2. 1.2.1  Quality  Data  Sources 

Developing  the  methodology  for  assessing  a  vendor's  quality 
performance  initially  concentrated  on  identifying  data  sources 
which  were  both  accessible  and  fairly  standardized  among  DLA' s 
Supply  Centers.  An  earlier  feasibility  study,  which  was 
conducted  by  DLA-DORO,  identified  three  primary  sources  of  data 
which  could  be  used  for  assessing  quality  performance  of  DLA' s 
vendors.  These  sources  are:  the  Customer /Depot  Complaints 
System  (CDCS) ,  Quality  Evaluation  Program  (QEP) ,  and  System  for 
Analysis  of  Laboratory  Testing  (SALT)  data  files.  These  files 
contain  the  data  needed  to  calculate  both  Center  and  FSC  quality 
scores  for  all  vendors  and  to  produce  an  archive  transaction 
file  for  vendor  viewing. 

As  the  file  name  indicates,  the  CDCS  file  contains  information 
relating  to  customer  generated  reports  that  reflect  the  quality 
of  products  issued  to  them.  Such  reports/complaints  are  also 
generated  internally  by  depot  and  supply  center's  Quality 
Assurance  representatives  when  they  determine  an  item  did  not 
meet  the  established  quality  standards.  Submitted  complaints  are 
reviewed,  investigated,  and  the  cause  is  determined.  Based  on 
guidance  provided  by  the  Study  Advisory  Group  (SAG) ,  this  file  is 
screened  to  capture  only  closed  complaints  in  which  a  vendor  was 
found  to  be  in  noncompliance  and  that  the  complaint  was  quality 
or  packaging  related.  The  SAG  members  also  recommended  shipping 
and  transportation  related  complaints  not  be  used  in  assessing  a 
vendor's  quality  performance,  since  a  vendor  could  not  be  held 
solely  accountable  for  such  shortcomings. 
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The  QEP  file  is  used  primarily  by  the  contracting  community  to 
summarize  a  vendor's  product  quality  history.  After  careful 
review  and  evaluation,  the  SAG  determined  that  the  QEP  file  would 
have  little  value  in  directly  determining  the  quality  performance 
of  vendors.  However,  the  SAG  did  feel  that  several  of  the  QEP 
data  elements  could  be  incorporated  into  DVRS  as  supplemental 
information.  Information  on  results  from  first  article  tests, 
pre-award  surveys,  post  award  evaluations,  quality  audits,  and 
contract  waivers  and  deviations  will  be  available  in  DPACS  should 
the  buyer  desire  additional  background  history. 

The  SALT  file  is  a  DLA  Directorate  of  Quality  Assurance 
(DLA-Q)  initiated  and  monitored  consolidated  database  which 
contains  testing  results  from  DLA's  Laboratory  Testing  Program. 
The  focus  of  the  query  into  this  file  is  to  ascertain  the 
percentage  and  the  degree  to  which  items  did  not  conform  to 
design  specifications.  SALT  data  are  linked  to  the  CDCS  file 
since  the  center's  quality  assurance  personnel  are  required  to 
generate  a  CDCS  complaint  whenever  an  item  fails  a  laboratory 
test.  As  discussed  later,  this  linkage  plays  a  critical  role  in 
identifying  quality  indicators. 

2. 1.2. 2  Quality  Indicator  Methodology 

Quality  Indicator  Selection:  Based  on  guidance  provided  by  the 
SAG's  Quality  Assurance  representatives,  three  DVRS  quality 
indicators  were  identified.  Selected  indicators  are  product 
complaints,  packaging  complaints,  and  laboratory  test  results. 
Product  and  packaging  complaints  are  selected  from  the  CDCS  by 
document  type  codes  and  discrepancy  codes.  Lab  test  failures  are 
linked  between  the  SALT  and  CDCS  by  the  CDCS  control  number  to 
provide  necessary  data  to  evaluate  the  test  results.  All 
complaints  and  test  failures  must  be  specifically  coded  as  a 
contract  nonconformance.  Also,  all  records  must  be  coded  as 
closed,  indicating  an  investigation  has  been  completed  and  the 
contractor  has  been  notified.  Complaint  indicators  are 
normalized  for  contract  volume  since  a  contractor  with  more 
contracts  has  more  opportunity  for  complaints  than  a  low  volume 
contractor . 

Vendor  Discrepancy  Volume  Determination  for  Product  and 
Packaging  Complaints:  The  first  step  in  developing  the  required 
quality  database  is  to  screen  the  CDCS  file  for  only  closed 
complaints  that  are  determined  to  be  the  fault  of  the  vendor. 
These  records  are  identified  by  having  a  cause  code  of  CN  or  CS 
(which  indicates  the  complaint  is  due  to  contractor  non- 
compliance)  and  having  a  date  entered  in  the  closed  date  field. 
Complaints  are  further  screened  to  capture  only  the  most 
recent  2  1/2  years  of  data  and  to  eliminate  all  remaining 
complaints  in  which  the  Commercial  and  Government  Entity  (CAGE) 
code  or  Procurement  Instrument  Identification  Number  (PUN)  are 
missing.  The  next  screen  is  to  eliminate  all  shipping  and 
transportation  type  complaints.  This  is  accomplished  by  keying 
on  both  the  Document  Type  and  Discrepancy  codes.  The  final  step 
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in  the  screening  process  is  to  eliminate  duplicate  complaints  for 
the  same  Contract  Line  Item  Number  (CLIN) .  Complaints  that  clear 
all  screening  criteria  are  then  written  to  a  quality  transaction 
file. 

The  second  step  is  to  append  laboratory  testing  information  to 
the  appropriate  transaction  record.  This  step  is  accomplished  by 
linking  SALT  records  to  the  transaction  file  by  matching  CDCS 
control  numbers  in  both  files.  SALT  data  appended  to  transaction 
file  include:  test  date,  laboratory  test  control  number,  number 
of  parts  tested,  number  of  critical  failure  parts,  number  of 
major  failure  parts,  and  number  of  minor  failure  parts. 

The  third  step  in  the  process  uses  transactional  file  records  to 
develop  a  by-vendor  and  FSC  frequency  count  on  the  number  of 
product  and  packaging  complaints  and  lab  test  failures.  For  the 
product  and  packaging  indicators  this  is  accomplished  by  keying 
on  the  Document  Type  and  Discrepancy  codes.  See  Appendix  A  for 
breakout  of  codes  in  determining  the  type  of  complaint. 

Development  of  Six  2-vear  Windows:  The  SAG  felt  that  the  trend 
of  DVRS  scores  could  be  important  in  bid  evaluation.  To  conduct 
trend  analysis  of  a  vendor,  six  2 -year  windows  are  developed. 

Each  window  is  off-set  by  1  month,  starting  with  the  most  recent 
2  years  of  history.  This  approach  provides  a  6  month  trend  for 
vendors'  performance.  The  close  date  for  each  valid  record  is 
used  to  determine  in  which  of  2-year  windows  a  complaint  should 
be  counted.  Computer  code  is  provided  to  take  advantage  of  the 
computer's  system  date  to  automatically  revise  the  window 
thresholds  during  each  update  run  of  the  model.  After  lengthy 
discussions,  the  SAG  came  to  an  agreement  that  the  Quality 
indicators  should  be  based  on  the  most  recent  2  years  of  data. 

The  decision  to  limit  the  data  to  2  years  was  based  on  the 
following: 

(1)  Compatible  time  period  with  the  Delivery  Indicator 
window. 

(2)  A  workable  time  frame  for  validating  past  complaint 
data. 

(3)  Reasonable  length  of  time  for  holding  a  vendor 
accountable  for  his  quality  performance. 

(4)  Reasonable  length  of  time  given  that  there  is  not 
always  an  exact  correlation  between  the  time  when  a 
complaint  was  generated  and  when  the  item  was  provided 
by  the  vendor. 

FSC  Indicator  Scores  For  Vendors:  Once  complaint  frequencies  are 
calculated  for  each  vendor  within  an  FSC,  the  next  step  accounts 
for  a  vendor's  contract  volume  within  that  FSC.  This  is  done  to 
provide  some  perspective  when  comparing  complaint  volume  between 
vendors.  The  selected  approach  is  to  develop  a  ratio  of 
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complaint  volume  over  contract  volume  for  a  vendor  within  an  FSC 
for  a  2-year  window.  The  Product  and  Packaging  scores  are: 


Product  Score  =  100  *  1-1 


#  of  Vendor  Quality  Ccnplairrts  within  FSC 

#  Vendor  CTJNS  Awarded  within  FSC 


— 


AND 


Packaging  Scare  =  100  *  1 


1 

1 
•  m 


#  of  Vendor  Packaging  Caqplaints  within  FSC 


#  Vendor  CI.TNS  Awarded  within  FSC 


:] 


The  vendor's  contract  data  is  obtained  by  matching  the  vendor's 
quality  count  file  to  the  delivery  indicator  file  which  contains 
the  contract  volumes  for  each  vendor  per  FSC.  A  rate  of  100.0  is 
provided  in  those  cases  in  which  a  vendor  has  no  complaint  data 
for  given  indicator.  In  cases  where  a  vendor  has  more  complaints 
than  lines,  the  model  gives  a  rate  of  zero. 


Laboratory  Testing  Indicator:  Laboratory  testing  indicator  rates 
are  determined  for  CDCS  records  that  have  a  nonblank  Test  Date 
field.  Here,  contract  volume  is  not  considered.  Lab  test 
failures  are  normalized  by  the  total  number  of  parts  tested. 
Laboratory  testing  scores  are  calculated  using  the  following 
equation: 


Lab  Testing 
where : 

CRTW 

MAJW 

MINW 

PCF 

PMAJF 

PMINF 

TPT 


Score  =  100  *  [1-  (CRTW  *  (PCF  /  TPT)  +  MAJW  * 

(PMAJF  /  TPT)  +  MINW  *  (PMINF  /  TPT) ) ] 

=  Critical  Failure  Weight 
=  Major  Failure  Weight 
=  Minor  Failure  Weight 

=  Total  #  Parts  within  an  FSC  with  Critical  Failures 

=  Total  #  Parts  within  an  FSC  with  Major  Failures 

=  Total  #  Parts  within  an  FSC  with  Minor  Failures 

=  Total  #  Parts  Tested 


The  weighting  values  approved  by  the  SAG  are: 

Critical  Failure  Weight  =  1.00 
Major  Failure  Weight  =  .80 

Minor  Failure  Weight  =  .40 

These  weighting  values  are  hard  coded  within  the  DVRS  model  and 
cannot  be  changed  by  the  buyers. 

Center  Level  Indicator  Scores  For  Vendors:  DVRS  quality  ratings 
are  also  computed  at  the  DLA  Supply  Center  (DSC)  level.  The 
equations  for  Center  level  ratings  are  essentially  identical  to 
those  used  to  compute  FSC  ratings.  The  only  difference  is  that 
the  data  is  expanded  from  FSC  to  the  vendor's  complete  data 
across  all  FSCs  managed  by  the  Center.  Center  level  indicator 
scores  are  determined  by  accumulating  a  vendor's  complaint  volume 
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across  all  FSCs.  These  scores  are  used  to  provide  the  buyer  an 
assessment  of  a  vendor's  performance  when  a  vendor  has  no 
contract  data  within  an  FSC  or  when  the  buyer  would  like  to 
determine  the  vendor  performance  across  the  center. 

Center  Mean  and  Standard  Deviation  Determination:  The  last  step 
calculates  a  center  level  mean  and  standard  deviation  for  each 
indicator.  These  statistics  are  needed  in  calculating  a  vendor's 
final  DVRS  score  (Section  2.1.4).  Product  and  packaging 
indicator  means  are  determined  by  dividing  the  accumulated 
quantity  of  complaints  for  each  indicator  type  by  the  total 
contract  volume  for  the  center  for  each  2-year  window.  The 
indicator  mean  equations  are: 

Tot.  #  of  Product  Complaints  §  Center 

Product  Indicator  Mean  =  _ 

Tot.  Contract  Lines  for  Center 

AND 

Tot.  #  of  Packaging  Complaints  §  Center 

Packaging  Indicator  Mean  =  _ 

Tot.  Contract  Lines  for  Center 

The  Laboratory  Test  Mean  is  calculated  in  a  slightly  different 
manner . 


j^Lab  Test  Scores 

Laboratory  Test  Mean  -  _ 

Tot .  DVRS  Records 


Each  indicator's  standard  deviation  value  (<r)  is  developed  using 
the  following  formula: 


where ; 


u 

N 

X 


Indicator  Mean 

Total  #  Vendor/ FSC  combinations 

Indicator  Score  for  Indicator/FSC  combination 


2.1.3 


DELIVERY  VENDOR  RATINGS 


2. 1.3.1 


Delivery  Data  Sources 


All  necessary  delivery  data  is  extracted  from  three  input  files: 
the  active  contract  file  (ACF) ,  DORO's  archived  closed  and  open 
contract  file  (ALLACF) ,  and  a  file  containing  up  to  the  most 
recent  6  months  of  closed  contracts  (USPF0203) .  These  files 
contain  the  data  needed  to  calculate  both  DSC  and  FSC  delivery 
scores  for  all  vendors  and  to  produce  an  archived  transaction 
file  for  vendor  viewing. 
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2. 1.3. 2 


Delivery  Indicator  Methodology 


Delivery  Indicator  Selection:  Two  DVRS  delivery  indicators  were 
identified  by  the  SAG.  The  first  indicator  is  the  percent  of 
vendor  contract  lines  that  were  delivered  on-time.  The  second 
delivery  indicator  is  associated  with  the  average  number  of  days 
a  vendor's  contract  lines  were  late  arriving.  This  indicator 
becomes  relevant  when  a  vendor's  contracts  are  late  and  is 
incorporated  within  DVRS  to  provide  the  buyers  insight  on  the 
degree  of  lateness,  given  that  contracts  are  late.  As  an  example, 
if  a  vendor  has  delivered  ten  contract  lines  in  the  past  2  years 
and  three  of  those  lines  were  delivered  late,  then  the  average 
days  late  is  calculated  using  only  those  three  late  contract 
lines. 

Procedures  for  Screening  Delivery  Data:  Before  making  any 
calculations,  the  data  must  first  pass  through  several  screening 
procedures.  First,  all  duplicate  CLIN  records  are  dropped  (only 
the  most  current  of  the  duplicate  CLINs  is  kept) .  Next,  each 
record,  which  represents  a  contract  line,  is  flagged  to  identify 
which  of  the  six  2 -year  windows  that  the  contract  lines  are 
associated  with  (see  Section  2. 1.2. 2).  This  is  done  by  keying  on 
the  contract  delivery  date  field  and  comparing  that  date  to  the 
opening  and  closing  date  of  the  2-year  window. 

Vendors'  contract  lines  are  considered  late  either  when  the 
material  arrives  late  or  when  the  ship  quantity  is  less  than  the 
contract  quantity  by  a  value  which  is  greater  than  what  is 
permissible  by  the  quantity  variance  code. 

The  percentage  of  on-time  deliveries  is  calculated  from  the 
number  of  times  a  contractor  ships  on  time  (or  the  quantity 
varies  within  authorized  limits)  to  total  lines.  Each  record 
(contract  line)  can  have  at  most,  one  late  frequency  count 
charged  against  it.  There  are  no  penalties  or  benefits  for  early 
deliveries  in  the  DVRS  scoring  methodology.  Thus,  early  contract 
deliveries  are  considered  on  time. 

There  are  some  exceptions  to  a  late  contract  line  which  are  used 
to  excuse  the  vendor.  For  instance,  if  the  Project  Action  Code 
equals  R5  or  if  the  Reason  For  Delay  Code  is  equal  to  H2,  H3, 

H4,  H5 ,  PI,  P2 ,  P3 ,  P4 ,  VI,  V2 ,  V3  ,V4,  V5,  or  V6,  or  if  the 
Termination  Code  equals  G  or  P,  the  contract  line  is  considered 
delivered  on  time.  In  each  of  the  above  cases,  it  was  judged  by 
the  SAG  that  the  contractor  was  not  responsible  for  the  late 
delivery.  Explanation  of  these  codes  is  found  in  Appendix  B. 

Delivery  Statistics:  After  conducting  the  required  screening 
procedures,  both  delivery  indicators  are  calculated  for  each 
vendor  for  the  entire  DSC.  The  percentage  of  contract  lines 
delivered  on-time  per  vendor  is  simply  found  by  dividing  the 
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number  of  lines  delivered  on-time  (where  the  ship  date  was  less 
than  or  equal  to  the  contract  delivery  date)  by  the  total  number 
of  contract  lines  for  that  vendor.  The  average  numbers  of  days 
late  is  calculated  by  dividing  the  total  sum  of  the  number  of 
days  late  by  the  number  of  late  contract  lines.  The  data 
generated  in  these  steps  is  then  used  to  determine  the  DVRS  score 
for  each  vendor  at  the  DSC  level.  The  same  procedures  are 
applied  to  calculate  similar  statistics  for  each  vendor  at  the 
FSC  level.  Vendor's  FSC  contract  frequency  data  are  determined 
by  simply  distributing  the  DSC  statistics  to  the  appropriate  FSC. 
This  is  accomplished  by  keying  in  on  the  first  four  positions  of 
the  contract  record  National  stock  Number  (NSN)  field.  Part 
numbered  buys  are  included  in  the  DSC  statistics  but  are 
unattributable  to  an  FSC. 

The  statistics  calculated  in  the  previous  steps  are  compared 
against  the  overall  DSC  averages  to  determine  the  actual  scores. 
The  next  step  in  the  process  is  to  calculate  an  overall  DSC 
average  percent  on-time,  average  days  late,  and  a  corresponding 
standard  deviation  for  each  of  the  six  2-year  windows.  The 
average  days  late  is  converted  into  a  "percent"  score  by 
subtracting  the  average  days  late  from  100.  If  the  average  days 
late  is  greater  than  100,  the  percent  is  converted  to  zero. 

2.1.4  THE  DVRS  SCORING  METHODOLOGY 

Next,  the  quality  indicator  information  at  the  DSC  level  is 
merged  with  the  delivery  indicator  at  the  DSC  level.  These  two 
files  are  matched  by  CAGE  to  form  a  single  file  containing  both 
delivery  and  quality  information.  Similarly,  the  FSC  indicator 
data  for  both  delivery  and  quality  are  merged  into  a  single  file. 
The  overall  DSC  indicator  averages  with  their  corresponding 
standard  deviations  are  merged  into  a  single  file. 

The  next  phase  calculates  the  delivery  score,  quality  score,  and 
total  combined  DVRS  score  at  the  DSC  and  FSC  level  for  all 
vendors.  Scores  are  calculated  in  the  following  manner:  First 
the  "range"  of  each  indicator  is  determined  by  subtracting  each 
lower  bound  from  each  upper  bound.  The  lower  bound  is  equal  to 
the  overall  DSC  indicator  average  minus  three  times  its  standard 
deviation.  The  upper  bound  is  equal  to  the  overall  DSC  indicator 
average  plus  three  standard  deviations.  If  any  lower  bound  value 
falls  below  zero,  that  lower  bound  is  then  truncated  to  zero.  If 
any  upper  bound  is  greater  than  100  percent,  that  upper  bound  is 
also  truncated  to  100  percent.  The  greatest  range,  between  the 
lower  and  upper  bounds,  is  used  to  calculate  the  scores  for  all 
five  indicators.  Each  indicator  is  then  normalized  to  this 
largest  range. 

The  normalization  process  is  accomplished  by  first  noting  that 
each  indicator  percent  lies  somewhere  within  its  own  indicator 
range.  The  position  on  this  range  is  transformed  to  a  similar 
position  on  the  largest  range.  In  most  cases,  the  range  will  be 
between  0  and  100  and  normalization  is  unnecessary.  For  example. 
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suppose  the  range  for  lab  testing  is  15  with  a  lower  bound  of  85 
and  an  upper  bound  of  100.  Further,  suppose  that  the  vendor  of 
interest  has  a  lab  testing  success  rate  of  97  percent.  Since  97 
is  80%  of  the  distance  from  85  to  100,  then  80  becomes  the  score 
of  that  indicator.  For  this  example,  the  vendor's  laboratory 
test  score  would  be  80.  The  formula  used  in  scoring  each  of  the 
DVRS  indicators  is: 

Indicator  Score  =  (indicator  %  -  L)*(max  -  min)/(U  -  L)  +  min 
where ; 

max  =  the  upper  bound  of  the  greatest  range  (usually  100) , 

min  =  the  lower  bound  of  the  greatest  range  (usually  0) , 

U  =  the  upper  bound  of  the  indicator, 

L  =  the  lower  bound  of  the  indicator. 

After  calculating  all  five  individual  indicator  scores,  the 
scores  are  grouped  to  yield  an  overall  delivery  score,  an  overall 
quality  score,  and  the  total  DVRS  score  for  each  vendor  at  both 
the  FSC  and  DSC  levels.  An  external  file  is  read  by  the  DVRS 
program  to  supply  weights  to  each  indicator  for  combining  into 
both  delivery  and  quality  score.  Weights  are  also  provided  to 
calculate  the  total  DVRS  score.  The  default  value  of  these 
weights  were  determined  by  the  SAG,  but  the  model  is  structured 
to  allow  the  centers  (the  DVRS  administrator)  to  change  overall 
weighting  values.  The  initial  weight  settings  are: 

The  default  values  of  the  overall  scoring  weights  are: 

Delivery:  50% 

Quality  :  50% 

The  delivery  indicator  weights  are: 

Percent  on-time:  60% 

Average  days  late:  40% 

The  quality  indicator  weights  are: 

Product:  50% 

Packaging:  20% 

Laboratory  tests:  30%. 

Using  these  weights,  the  following  formulas  are  used  to  generate 
the  quality  and  delivery  scores  for  FSCs  and  DSCs: 

Quality  score  =  (PW  *  Prod)  +  (PRW  *  Pack)  +  (LW  *  Lab) 
where ; 

PRW  =  packaging  weight, 

PW  =  product  weight, 

LW  =  lab  testing  weight. 
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Prod  =  product  indicator  score. 

Pack  =  packaging  indicator  score, 

Lab  =  lab  testing  indicator  score. 

A  vendor's  overall  delivery  score  is  calculated  using  the 
following  equation: 

delivery  score  =  (OW  *  Perc)  +  (AW  *  Avg) 


where ; 

OW  =  percent  on-time  weight, 

AW  -  average  days  late  weight, 

Perc=  percent  on-time  indicator  score, 

Avg  =  average  days  late  indicator  score. 

Once  the  quality  and  delivery  scores  are  calculated,  the  overall 
DVRS  score  is  determined  by: 

DVRS  score  =  (QW  *  quality  score)  +  (DW  *  delivery  score) 


where ; 


QW  =  quality  score  weight, 

DW  =  delivery  score  weight. 

The  last  step  in  the  DVRS  methodology  involves  an  examination  of 
vendors  that  have  changed  their  CAGE  codes  within  the  past  2 
years.  If  DVRS  determines  that  a  CAGE  has  changed  within  that 
time  frame,  all  scores  under  the  prior  CAGE  are  be  combined  with 
the  scores  of  the  new  CAGE  (unless  the  vendor  has  shown 
sufficient  cause  not  to  combine  the  scores) .  These  scores  are 
combined  using  a  weighted  average  based  on  the  number  of  contract 
lines  for  each  CAGE.  Due  to  the  nature  of  the  CAGE  file,  only 
one  previous  CAGE  can  be  identified. 

2.2  VALIDATION  METHODOLOGY 

The  DVRS  prototype  was  tested  in  a  variety  of  ways.  First,  the 
model  was  run  at  IPCR  using  live  production  files.  An  ADP 
verification  was  made  to  insure  the  correct  data  files  were 
captured  and  processed  through  DVRS,  yielding  the  analytically 
correct  ratings  based  on  the  methodology  described  in  Section 
2.1.  Secondly,  DVRS  ratings  were  compared  to  other  sources  of 
information  on  contractor  performance  to  check  for  consistency. 
Finally,  a  functional  test  is  planned  at  DGSC  where  DVRS  ratings 
will  be  used  by  Contracting  personnel  for  a  period  of  time  on 
certain  FSCs.  The  results  of  the  functional  test  will  not  be 
known  until  early  FY  93  and  will  be  reported  in  a  separate 
document . 

ADP  validation  and  verification  was  conducted  by  checking  input 
and  output  files  for  each  step  in  the  28  step  program.  Values 
were  checked  to  make  sure  data  did  not  fall  outside  theoretical 
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limits  (such  as  negative  numbers) .  For  several  FSC/ contractor 
combinations,  raw  data  was  manually  reviewed  along  each  step  to 
verify  data  screening  and  insure  that  final  ratings  matched 
manual  calculations. 

To  check  the  consistency  of  DVRS  ratings,  a  Mann-Whitney 
nonparametric  test  was  performed  to  assess  the  degree  of 
agreement  between  DVRS  and  other  data  sources.  A  list  of 
contractors  with  known  excellent  performance  was  obtained  from 
each  hardware  center's  representatives  to  the  SAG.  These  were 
generally  from  the  Quality  Vendor  Program  (QVP)  list.  A  list  of 
poor  performers  was  also  obtained  from  the  SAG  representative. 
These  lists  were  compared  with  DVRS  scores  to  measure  the  ability 
of  DVRS  to  distinguish  between  known  poor  and  excellent 
performers.  A  hypothesis  test  was  constructed  at  the  95  percent 
confidence  level.  The  design  of  the  test  was  to  identify  each 
vendor  as  either  a  high  or  low  performer  and  then  sort  the 
contractors  in  DVRS  sequence,  low  to  high.  The  rank  of  each 
contractor  in  the  sample  was  then  determined  (worst  DVRS  rating 
rank  is  1,  second  worst  rank  is  2,  etc.).  The  sum  of  ranks  for 
the  high  performers  was  then  compared  to  the  sum  of  ranks  for  low 
performers.  A  null  hypothesis  was  established  that  the  rank  sums 
were  not  statistically  different  (i.e.  high  performers  and  low 
performers  were  not  differentiated  by  DVRS) .  The  alternate 
hypothesis  was  that  high  performing  vendors  had  higher  DVRS 
ratings  than  low  performers.  If  the  rank  sums  of  the  two  groups 
were  fairly  close,  the  null  hypothesis  would  be  accepted.  If  the 
rank  sums  of  the  high  performers  are  much  greater  than  those  of 
poorer  performers,  the  null  hypothesis  would  be  rejected  and  the 
alternate  would  be  accepted. 

Initially,  DVRS  will  be  available  at  DGSC  for  three  FSCs.  Those 
selected  were  FSC  5975  -  Electrical  hardware  and  supplies; 

FSC  9330  -  Plastics  fabricated  materials;  and,  FSC  9390  - 
Miscellaneous  fabricated  nonmetalic  materials  ( asbestos , cork , 
clay,  etc) .  An  effort  was  made  to  manually  review  the  historical 
data  on  those  FSCs  and  to  correct  errors  prior  to  implementation. 

2.3  IMPLEMENTATION  CONCEPT 

DVRS  is  designed  to  run  monthly  on  the  IPCR  mainframe  computer. 
Each  month  at  a  scheduled  time,  DVRS  will  capture  the  current 
information  in  the  SAMMS  production  files  and  augment  the  data 
with  the  SALT  file  and  DIDB  data  to  produce  DVRS  scores.  Ratings 
by  CAGE  and  FSC,  along  with  summary  ratings  at  the  center  level, 
associated  statistics  and  raw  data  will  be  stored  on  tape  and 
archived.  DVRS  ratings  and  associated  statistics  will  be 
downloaded  from  the  mainframe  to  the  Distributed  Minicomputer 
System  (DMINS) .  DPACS  will  load  the  DMINS  files  into  UNIFY 
databases  which  will  feed  various  DPACS  screens.  Buyers  and 
contracting  officers  will  access  the  ratings  through  a  variety  of 
menu  driven  options  on  their  personal  computers.  Archived  data 
will  eventually  be  sent  to  a  proposed  DLA  bulletin  board  that 
will  be  designed  for  contractor  access. 
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There  are  two  manual  processes  built  into  DVRS.  When  a  vendor 
reviews  its  own  archived  data  and  does  not  agree  with  the 
accuracy  of  the  data,  the  vendor  must  notify  the  DSC.  DVRS 
contains  a  data  element  for  each  CAGE/FSC  combination  that  can  be 
manually  set  as  a  Y/N  (Yes/No)  indicator  to  tell  DPACS  that  the 
vendor  disputes  the  rating.  Secondly,  if  a  vendor  believes  that 
performance  under  a  prior  CAGE  code  should  not  be  used  to  measure 
the  current  CAGE ' s  performance,  a  similar  Y/N  flag  is  set  to 
suppress  combining  of  scores. 

Central  to  the  design  of  DVRS  is  the  need  to  control  and  monitor 
DVRS  applications.  A  DVRS  focal  point  is  needed  to  manage  the 
manual  processes  within  DVRS,  coordinate  activities  between  the 
contracting  officer,  contractor,  IPCR,  DVRS  system  developers, 
DVRS  system  maintainers,  and  other  Supply  Center  organizations. 
Additionally,  the  DVRS  focal  point  will  provide  required  reports 
to  higher  level  DLA  activities.  This  proposed  focal  point  has 
been  designated  as  DVRS  Administrator  for  the  DSC.  DLA-PPR  and 
the  Office  of  Civilian  Personnel  have  developed  Agency  Job 
Guidelines  to  establish  series,  grade,  and  staffing  requirements 
for  this  function. 
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SECTION  3 
FINDINGS 

3.1  MODEL  RESULTS 

DVRS  was  run  on  the  IPC-Richmond  mainframe  computer  during  the 
month  of  March  1992.  Total  run  time  was  approximately  7  CPU 
minutes  on  an  Amdahl  5880.  Data  checks  confirmed  that  all 
production  files  were  accessed  correctly  and  that  data 
integration  from  all  sources  occurred  as  planned.  Computations 
were  hand  checked  to  confirm  internal  processes  correctly  compute 
vendor  ratings.  Table  1  summarizes  the  ratings,  highlighting 
those  FSCs  that  were  chosen  for  functional  testing.  Data  from 
these  FSCs  were  undergoing  manual  review  and  correction  at  the 
time  of  processing  and  are  probably  more  accurate  than  data  for 
other  FSCs.  The  high  quality  ratings  are  partially  explained  by 
the  fact  that  the  Lab  Testing  program  at  DGSC  is  in  its  infancy 
and  therefore  almost  no  lab  test  data  was  available.  When  lab 
testing  matures,  the  DVRS  quality  ratings  may  be  much  lower  and 
comparable  to  Delivery  Scores.  Output  products,  i.e.  the  vendor 
rating  files  and  archive  transaction  files  were  reviewed  to 
insure  completeness  and  accuracy.  Vendor  ratings  were  downloaded 
successfully  to  the  DMINS  system  and  are  available  for  use  by 
DPACS.  Unforeseen  problems  with  DPACS  have  delayed  the 
integration  of  DVRS  into  DPACS  by  the  DPACS  developer.  Partial 
integration  is  now  planned  for  September  1992  with  full 
integration  by  January  1993. 

3.2  VALIDATION  RESULTS 

In  addition  to  insuring  that  the  data  was  processed  correctly, 
statistical  tests  were  conducted  to  verify  that  DVRS  ratings 
matched  the  perception  of  experts  or  other  sources.  A  reality 
check  was  conducted  to  compare  a  sample  of  contractor  ratings  for 
those  vendors  that  have  a  known  track  record.  By  nature  of  the 
business  practices  used,  contracting  shops  in  DLA  have  hard  data 
visibility  only  for  extremely  good  and  extremely  poor  performers. 
Thus,  DVRS  ratings  for  those  high  and  low  performing  vendors  were 
checked  to  insure  that  the  ratings  worked  at  the  extremes  of  the 
distributions  of  vendors.  This  test  did  not  measure  DVRS  ability 
to  distinguish  between  vendors  of  moderate  performance  history. 
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QVP  89.0  81.3  96.7  16  24963 


Lists  of  high  and  low  performers  were  provided  by  SAG  members 
from  DGSC1,  DCSC,  and  DISC.  For  DGSC,  the  FSC  was  also 
identified.  Therefore,  the  list  was  matched  to  the  DVRS  rating 
of  the  vendor  for  the  particular  FSC.  For  the  other  centers,  no 
FSC  designation  was  available;  for  these  vendors,  the  center 
average  DVRS  rating  was  used.  For  DCSC  and  DISC,  no  quality 
assurance  data  was  available,  so  the  DVRS  rating  used  was  the 
delivery  rating.  Raw  delivery  data  from  DGSC  was  current;  but 
data  from  the  other  centers  was  only  available  through  September 
1991. 

To  illustrate  the  use  of  the  Mann-Whitney  nonparametric  test  (see 
para  2.2),  the  DISC  data  will  be  highlighted.  Test  data  for  DGSC 
and  DCSC  is  shown  in  Appendix  C.  In  April  1992,  DISC  provided 
the  CAGE  codes  for  10  vendors,  five  each  high  and  low  delivery 
performers.  DVRS  delivery  ratings  were  produced  and  are  shown  in 
Table  2  in  order  of  lowest  to  highest  rating.  The  rank  sum  of 
each  group  (high  and  low  performers  as  identified  by  the  experts) 
is  the  sum  of  the  rankings  with  rank  1  meaning  the  lowest  DVRS 
rating  and  10  the  highest  rating. 


TABLE  2 

Mann-Whitney  Test  Data  -  DISC 


Functional 

DVRS 

Expert 

Delivery 

Rank 

Contractor 

Ratincr 

Ratincr 

Rank 

Sum 

Z 

Low 

44 

1 

Y 

Low 

52 

2 

X 

Low 

58 

3 

W 

Low 

62 

4 

V 

Low 

63 

5 

E 

High 

70 

6 

15 

D 

High 

73 

7 

C 

High 

77 

8 

B 

High 

79 

9 

A 

High 

83 

10 

40 


1  "Poor”  performers  at  DGSC  were  those  that  applied  for  but 

failed  to  meet  Quality  Vendor  Program  criteria.  These  vendors 
were  not  necessarily  below  average  performers. 
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A  test  statistic  is  calculated  and  compared  to  a  limit  derived 
from  the  normal  distribution  to  determine  if  the  sample  results 
are  significant.  In  this  case,  the  Mann-Whitney  statistic  for 
the  low  performers  is  -2.51  compared  to  a  critical  value  of  -1.96 
for  a  two-tailed  test  at  95  percent  confidence.  From  Table  2  it 
is  apparent  that  there  are  no  inconsistencies  since  all  the  high 
performers  are  ranked  higher  than  all  the  low  performers.  The 
possibility  of  chance  correlation  to  this  degree  is  less  than  5 
percent  (1.2  percent  to  be  exact).  Therefore,  for  DISC,  it  is 
concluded  that  DVRS  ratings  can  distinguish  between  good  and  poor 
performers. 

The  sample  sizes  for  DGSC  and  DCSC  were  much  larger  than  for 
DISC.  Although  there  were  cases  at  these  centers  where 
occasionally,  an  identified  low  performer  rated  highly  in  DVRS 
and  vice  versa,  the  correlation  achieved  using  Mann-Whitney  tests 
combined  with  the  large  sample  produced  overwhelming  results 
leading  to  the  same  conclusion.  Table  3  summarizes  test  results. 
In  all  cases,  the  DVRS  passed  the  test  with  95  percent 
confidence. 


TABLE  3 

Mann-Whitney  Results 


Delivery  (D) 

Z 

or 

Man**  .  ..xtrey 

Type  I 

Center 

SamDle  Size 

Test  Statistic 

Error 

DISC 

D 

10 

-2.51 

.0120 

DCSC 

D 

154 

-5.62 

<.0001 

DGSC 

D 

100 

-:.9i 

<.0001 

DGSC 

Q 

85 

-2.12 

.0348 
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SECTION  4 
CONCLUSIONS 


The  DLA  Vendor  Rating  System  prototype  model  has  proven  to  be  a 
"nearly-automated”  system  which  provides  measures  of  contractor 
past  performance  for  delivery  effectiveness  and  quality  assurance 
effectiveness.  The  prototype  has  been  run  in  the  production 
environment  on  the  IPC-Richmond  mainframe  using  DGSC  SAMMS  data, 
DIDB  data  and  lab  test  data  transferred  from  the  DLA 
Administrative  Support  Center.  Mainframe  processes  have  been 
validated  and  statistical  tests  were  successfully  conducted  to 
assure  measures  were  reasonable. 

The  ability  to  provide  these  measures  to  the  buyers  for 
transition  from  prototype  to  production  has  not  yet  been 
achieved.  The  insertion  of  DVRS  into  the  DPACS  system  has  been 
delayed  because  of  technical  problems  with  the 

DPACS  system.  With  minimal  programming  effort,  but  with  a  higher 
burden  on  users,  DVRS  could  be  transformed  into  a  stand-alone 
system,  independent  of  DPACS. 
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SECTION  5 
RECOMMENDATIONS 


It  is  recommended  that  the  DVRS  prototype  be  conditionally 
approved  for  transition  into  the  DLA  standard  system.  Final 
approval  should  be  contingent  on  the  development  of  systems  to 
allow  better  access  to  the  DVRS  by  buyers  and  vendors.  Buyer 
access  will  be  best  achieved  through  DPACS.  Vendor  access  will 
be  best  achieved  via  electronic  communication. 

It  is  recommended  that  the  DVRS  sponsors,  DLA-PPR  and  DLA-QL , 
sponsor  additional  projects  to  guide  the  transfer  of  DVRS  into  a 
standard  system.  These  projects  involve  coordination  with  system 
developers  for  buyer  and  vendor  access  and  model  modifications 
needed  for  use  of  DVRS  at  the  other  hardware  centers. 


5-1 


APPENDIX  A 

IDENTIFICATION  OF  DVRS  QUALITY  AND  PACKAGING  INDICATORS 

BY  CD CS  CODES 
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APPENDIX  A 

IDENTIFICATION  OF  DVRS  QUALITY  AND  PACKAGING  INDICATORS 

BY  CDCS  CODES 


CDCS  complaints  can  be  grouped  into  the  four  following 
categories:  Packaging,  Quality,  Shipping,  and  Transportation.  As 
mentioned  in  the  DVRS  main  report,  Shipping  and  Transportation 
type  complaints  are  screened  out  by  keying  on  both  the  CDCS 
Discrepancy  Code  and  the  Document  Type  Code  (ref  CDCS  Users 
Manual,  Working  Copy,  June  1986) .  The  screening  process  is  based 
on  using  the  following  code  combination  matrix. 

Note  that  Document  Types  of  5  and  8  are  not  included  in  this 
matrix  since  these  type  records  are  also  screened  out  of  the  DVRS 
model.  The  matrix  coding  scheme  is: 


CODE  TYPE  COMPLAINT 

P  Packaging 

Q  Quality  (Product) 

S  Shipping 

T  Transportation 


Document  Type 
3  4  6  7 


Q  Q  T  Q 


Discrepancy 


Code _ 0 

A1  Q 

A2  Q 

A3  Q 

A4  Q 

A5  Q 


Q  Q  T  Q  Q 

Q  Q  T  Q  Q 

Q  Q  T  Q  Q 

Q  Q  T  Q  Q 


9  A  B  C  B 
Q  Q  Q  Q  Q  Q 
Q  Q  Q  Q  Q  Q 
Q  Q  Q  Q  Q  Q 
Q  Q  Q  Q  Q  Q 
Q  Q  Q  Q  Q  Q 


Cl 

C2 

C3 

C4 

C5 

C6 


Q  Q  Q  T  Q  Q  Q 
Q  Q  Q  T  Q  Q  Q 
Q  Q  S  T  Q  S  S 
Q  Q  Q  T  Q  Q  Q 
Q  Q  T  T  Q  T  T 
Q  Q  T  T  Q  T  T 


Q  Q  Q  Q  Q 
Q  Q  Q  Q  Q 
S  Q  Q  Q  Q 
Q  Q  Q  Q  Q 
T  Q  Q  Q  Q 
T  Q  Q  Q  Q 


D1 

D2 

D3 


Q  Q  S  T  Q  S 
Q  Q  S  T  Q  S 
Q  Q  S  T  Q  S 


S  S  Q  Q  Q  Q 
S  S  Q  Q  Q  Q 
S  S  Q  Q  Q  Q 


LI 

L2 

L3 

L4 

L5 

L6 

L7 

L8 


Q  Q  Q  T  Q 
Q  Q  Q  T  Q 
Q  Q  Q  T  Q 
Q  Q  Q  T  Q 
Q  Q  Q  T  Q 
Q  Q  Q  T  Q 
Q  Q  Q  T  Q 
Q  Q  Q  T  Q 


Q  Q  Q 

Q  Q  Q 

Q  Q  Q 

Q  Q  Q 

Q  Q  Q 

Q  Q  Q 

Q  Q  Q 

Q  Q  Q 


Q  Q  Q  Q 
Q  Q  Q  Q 
Q  Q  Q  Q 
Q  Q  Q  w 
Q  Q  Q  Q 
Q  Q  Q  Q 
Q  Q  Q  Q 
Q  Q  Q  Q 


Ml 


Q  Q  S  T  Q  S  S 


S  Q  Q  Q  Q 
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Discrepancy 

Code _ 

Ol 

02 

03 

04 

PO 

PI 

P2 

P3 

P4 

P5 

P6 

P7 


Q1 

Q2 

Q3 

Q4 

Q5 

Q6 

Q7 

51 

52 

53 

54 

55 

T1 

T2 

T3 

T4 

T5 

T6 

WO 

W1 

W2 

W3 

W4 

W5 

W6 

W7 

W8 

W9 

XI 

XL 


0 _ 1 

Q  Q 
Q  Q 
Q  Q 
Q  Q 

Q  Q 
Q  Q 
Q  Q 
Q  Q 
Q  Q 
Q  Q 
Q  Q 
Q  Q 

Q  Q 
Q  Q 
Q  Q 
Q  Q 
Q  Q 
Q  Q 
Q  Q 

Q  Q 
Q  Q 
Q  Q 
Q  Q 
Q  Q 

Q  Q 
Q  Q 
Q  Q 
Q  Q 
Q  Q 
Q  Q 

Q  Q 
Q  Q 
Q  Q 
Q  Q 
Q  Q 
Q  Q 
Q  Q 
Q  Q 
Q  Q 
Q  Q 

Q  Q 
Q  Q 


Document  Type 
2  3  4  6  7  9 

S  T  Q  S  S  S 

S  T  Q  S  S  S 

S  T  Q  S  S  S 

T  T  Q  T  T  T 


P  T  P  P  P 

P  T  P  P  P 

P  T  P  P  P 

P  T  P  P  P 

P  T  P  P  P 

pTP  P  P 

FTP  P  P 

PTP  P  P 


P 

P 

P 

P 

P 

P 

P 

P 


Q  T  Q 
Q  T  Q 
Q  T  Q 
Q  T  Q 
Q  T  Q 
Q  T  Q 
Q  T  Q 

S  T  Q 
S  T  Q 
S  T  Q 
S  T  Q 
T  T  Q 

Q  T  Q 
Q  T  Q 
Q  T  Q 
PTP 
Q  T  Q 
PTP 


Q  Q  Q 

Q  Q  Q 

Q  Q  Q 

Q  Q  Q 

Q  Q  Q 

Q  Q  Q 

Q  Q  Q 

S  S  S 

s  s  s 

s  s  s 

s  s  s 

T  T  T 

Q  Q  Q 

Q  Q  Q 

Q  Q  Q 

P  P  P 

Q  Q  Q 

P  P  P 


Q  T  Q 
Q  T  Q 
Q  T  Q 
Q  T  Q 
Q  T  Q 
Q  T  Q 
Q  T  Q 
Q  T  Q 
Q  T  Q 
Q  T  Q 

Q  T  Q 
Q  T  Q 


Q  Q  Q 

S  S  S 

S  S  S 

Q  Q  Q 

Q  Q  Q 

Q  Q  Q 

Q  Q  Q 

Q  Q  Q 

Q  Q  Q 

Q  Q  Q 

Q  Q  Q 

Q  Q  Q 


A 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 


B. 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 

Q 


C _ g 

Q  Q 
Q  Q 
Q  Q 
Q  Q 

Q  Q 
Q  Q 
Q  Q 
Q  Q 
Q  Q 
Q  Q 
Q  Q 
Q  Q 

Q  Q 
Q  Q 
Q  Q 
Q  Q 
Q  Q 
Q  Q 
Q  Q 

Q  Q 
Q  Q 
Q  Q 
Q  Q 
Q  Q 

Q  Q 
Q  Q 
Q  Q 
Q  Q 
Q  Q 
Q  Q 

Q  Q 
Q  Q 
Q  Q 
Q  Q 
Q  Q 
Q  Q 
Q  Q 
Q  Q 
Q  Q 
Q  Q 

Q  Q 
Q  Q 
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PROJECT  ACTION  CODE 
REASON  FOR  DELAY  CODE 
TERMINATION  CODE 
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APPENDIX  B 


The  Project  Action  Code,  Reason  for  Delay  Code,  and  Termination 
Code  are  used  to  help  evaluate  a  late  delivery.  The  actual  codes 
used  in  the  screening  process,  along  with  their  references  and 
definitions,  are  listed  in  the  following  paragraphs. 

PROJECT  ACTION  CODE 

Reference:  DLAM  8100.1  2-23,  ACTION-DLA  Operational  Procedures 

and  Personnel  Computer  Users  Manual,  dated  7  Oct  88 

R5  -  Delivery  action  on  this  CLIN  has  been  suspended  due  to  a 
request  for  a  technical  or  legal  determination.  This  CLIN  may  be 
terminated,  cancelled,  or  extended  if  the  legal/technical 
decision  so  dictates. 

REASON  FOR  DELAY  CODE 

Reference:  DLAM  4715.1  I,  SAMMS  Manual  of  Operating  Procedures 

for  the  Contracting  Subsystem,  Vol  1,  dated  2  Jan  91 

H2  -  GFM/GFP  supplied  late. 

H3  -  GFM/GFP  inadequate. 

H4  -  GFM/GFP  defective. 

H5  -  GFM/GFP  uneconomical ly  repairable. 

PI  -  Natural  disaster,  fire. 

P2  -  Natural  disaster,  flood. 

P3  -  Natural  disaster,  extreme  weather. 

P4  -  Natural  disaster,  earthquake. 

VI  -  Energy  shortage,  natural  gas. 

V2  -  Energy  shortage,  electricity. 

V3  -  Energy  shortage,  coal. 

V4  -  Energy  shortage,  gasoline. 

V5  -  Energy  shortage,  fuel  oil  (all  types) . 

V6  -  Energy  shortage,  liquefied  gas  (all  types) . 

TERMINATION  CODE 

Reference:  DLAM  4715.1  1-2  A20-1,  SAMMS  Manual  of  Operating 

Procedures  for  the  Contracting  Subsystem,  Vol  1,  Parts  l  &  2, 
dated  2  Jan  91 

G  -  Termination  for  government  convenience. 

P  -  Pending  termination  for  government  convenience. 
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APPENDIX  C 

MANN -WHITNEY  STATISTICAL  TESTS 

Nonparametric  test  data  for  Mann-Whitney1  tests  are  contained 
in  this  Appendix.  Tests  for  delivery  ratings  by  FSC  at  DGSC  are 
provided  on  page  C-5.  Similar  quality  ratings  are  shown  on  page 
C-6.  Delivery  ratings  for  DCSC  are  shown  on  pages  C-7  through 
C-8 . 

The  objective  to  be  tested  is  to  determine  if  DVRS  can 
statistically  distinguish  between  known  poor  and  known  excellent 
performers.  If  there  is  no  ability  to  distinguish,  then  poor 
performers  and  good  performers  will  achieve  comparable  DVRS 
ratings.  When  DVRS  scores  of  a  sample  containing  both  types  of 
performers  are  ranked  ordered,  then  the  types  will  be  randomly 
arranged  if  there  is  no  effect.  If  DVRS  does  distinguish,  then 
the  poor  performers  will  be  clustered  at  the  low  range  of  DVRS 
scores  and  the  high  performers  will  tend  toward  the  upper  ranges 
of  scores.  If  we  define  M  as  the  median  value,  where  My  is 
the  median  value  for  high  performers  and  Mx  is  the  median 
value  for  low  performers,  the  hypothesis2  test  is  set  up  as 
follows: 

H0:  My  =  Mx 

hi:  *4  *  M* 

The  test  statistic,  which  is  compared  to  the  normal  deviate,  is 
(if  the  sample  size  is  large  and  there  are  few  ties) 

Tx-  m(N+l)/2 

Z  =  .  Where 

Ym  n(N+l)/12 

Tx  is  the  rank  sum  of  the  poor  performers 
m  is  the  number  of  poor  performers 
n  is  the  number  of  excellent  performers 
N  is  the  total  sample  size  (m+n) 

To  pass  the  test  at  95  percent  confidence,  the  above  equation 
must  be  less  than  -1.96  or  greater  than  +1.96.  If  the  test 
statistic  is  outside  these  limits,  we  reject  HQ  and  accept 
Hj_,  and  conclude  that  DVRS  does  indeed  distinguish  between 
poor  and  excellent  performers.  (Note:  This  test  could  be 
designed  as  a  one-tail  hypothesis  test,  in  which  case  the  limit 
is  less  than  -1.64.) 


1  Jean  Dickerson  Gibbons,  Nonparametric  Methods  for 
Quantitative  Analysis,  Second  Edition,  (American  Sciences  Press, 
Inc.),  pp.  159-171 

2  John  E.  Freund,  Modern  Elementary  Statistics,  3rd  Edition, 
(Printice  Hall  Inc.),  pp.  243-248 
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In  the  following  tables,  each  line  is  an  anonymous  contractor.  A 
MY"  in  the  last  column  reflects  that  the  Center  has  determined 
that  vendor  to  be  a  high  performer  and  a  ,,N"  designates  a  poor 
performer. 
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TABLE  C-l. 

Test  Data 

for 

Delivery 

Ratings  by 

FSC  at  DGSC 

? 5  c  cage 

U  Z  C.  U  P  E  _  t. 

FSC  CAGE 

OSCQRE  0 

1 

6  105 

4  3.9 

N 

5  1 

62  !  0 

96  .  1 

7 

2 

6220 

44.8 

N 

52 

68  1  0 

86 . 2 

t 

3 

5995 

46 . 8 

N 

53 

9  1  50 

86  .  3 

Y 

a 

6850 

49.2 

N 

54 

6685 

86  .  3 

1 

Y 

5 

6680 

53.1 

Y 

55 

59  7  5 

86.5 

1 

>■ 

6 

668  5 

55.5 

Y 

56 

7690 

35  .  5 

I 

y 

7 

24  29 

5o  .  5 

N 

57 

36  1  0 

86  .  ? 

y 

8 

0  66  5 

5c  . ; 

N 

58 

6  105 

8  7  .  C 

9 

6  130 

58.5 

N 

59 

1  560 

8  7.' 

10 

6  l  50 

58.6 

Y 

6  0 

5995 

8  7.6 

) 

1 

1  1 

7  32Q 

60  .  3 

N 

6  1 

5  9  c  n 

q  *  - 

12 

6230 

6  1.1 

Y 

62 

6  ■  C  5 

5  -  .  C 

1  3 

5970 

6  3.4 

Y 

6  3 

5  9  7  0 

Qo  ' 

J 

1  4 

6810 

6  3.8' 

N 

64 

6220 

89.3 

Y 

15 

■  7320 

63.8 

Y 

65 

6240 

89 . 8 

Y 

16 

5970 

65 . 3’ 

N 

66 

599  5 

8°  Q 

Y 

17 

2040 

6  7.0 

Y 

67 

4  1  40 

90 . 6 

Y 

18 

6685 

68 . 1 

Y 

68 

59  7  5 

90.  1 

Y 

19 

9330 

68.2 

N 

69 

6635 

90 . 4 

Y 

20 

6680 

68 . 2 

Y 

71 

9150 

90.5 

Y 

21 

6810 

69.4 

Y 

71 

5995 

90.6 

Y 

22 

6680 

69.4* 

Y 

72 

3439 

91.1 

Y 

23 

1560 

69.6 

M 

73 

5995 

91.4 

in 

24 

5995 

70 . 5 

N 

74 

5975 

91.9 

Y 

25 

6230 

70.5 

f 

75 

5970 

92 . 4 

Y 

26 

7.320 

7  1.1 

Y 

76 

6130 

92.8 

Y 

27 

3455 

7  1.8 

Y 

77 

6?30 

94 . 3 

Y 

28 

6210 

72.6 

Y 

78 

6105 

94 . 3 

Y 

29 

1680 

7  2.2 

Y 

79 

6750 

94 . 4 

Y 

30 

6750 

7  4.0 

Y 

80 

6695 

94  .  5 

Y 

31 

6680 

74.3 

Y 

81 

3439 

98  .  1 

Y 

32 

4240 

7  5.6 

N 

82 

3444 

98  .  1 

Y 

33,. 

6680 

75.9 

N 

83 

3439 

99 . 2 

Y 

34 

4140 

76.7 

Y 

84 

6230 

100.0 

Y 

35 

6105 

76.8 

Y 

85 

6210 

100.0 

Y 

36 

6850 

76.9 

N 

86 

5995 

100. 0 

Y 

37 

5977 

7  7.0 

Y 

87 

4130 

100.0 

Y 

38 

6260 

77.3 

Y 

88 

6660 

100.0 

Y 

39 

7320 

78 . 0 

N 

89 

1560 

100.0 

Y 

40 

6240 

78.2 

Y 

90 

6685 

100.0 

Y 

41 

6685 

78.7 

Y 

91 

3460 

100.0 

Y 

42 

6610 

79  .  1 

Y 

92 

7310 

100.0 

Y 

43 

597  5 

79  .  7 

Y 

93 

9330 

100.0 

Y 

44 

9130 

8  1.4 

Y 

94 

6105 

100.0 

Y 

45 

6  130 

8  1.6 

Y 

*  95 

7320 

100.0 

Y 

46 

6230 

92.3 

Y 

96 

6685 

100.0 

Y 

47 

664  5 

83.8 

N 

97 

9150 

100.0 

Y 

48 

59  7  5 

84.4 

Y 

98 

9330 

100.0 

Y 

49 

5995 

85.5 

Y 

99 

9150 

100.0 

Y 

50 

5995 

85.8 

Y 

100 

6150 

100.0 

Y 

T  = 

X 

1  +  2+3  +  4-1-7+8+9+11  +  14  +  16+19+23  +  24  +  32  +  33  +  3  6+39+47 

-71  =  401 

m  a  19  N  =  100 

n  =81  Z  =-4.907 
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TABLE  02.  Test  Data  for  Quality  Ratings  by  FSC  at  DGSC 


F SC  CAGE  QSCORE  0 


1 

6660 

34 . 4 

r 

2 

9150 

60 . 5 

Y 

3 

6105 

68 . 5 

Y 

4 

6810 

76.4 

N 

5 

6810 

83.6 

Y 

6 

6810 

85.2 

Y 

7 

3439 

86 . 3 

Y 

8 

6685 

87 . 5 

N 

9 

7320 

87.8 

Y 

10 

9330 

88 . 3 

Y 

1 1 

7320 

90.6 

Y 

12 

6130 

'  90.7 

Y 

13 

7320 

91.9 

Y 

14 

6210 

92.2 

Y 

15 

5970 

92.9 

Y 

16 

6260 

93.4 

Y 

17 

3460 

93.5 

Y 

18 

6150 

•*93.8 

Y 

19 

9150 

93.9 

N 

20 

1560 

94.9 

Y 

21 

6230 

95.5' 

Y 

22 

9150 

96. 1 

Y 

23 

1680 

96 . 2 

Y 

24 

3455 

96 . 8 

Y 

25 

4130 

97  .  1 

Y 

26 

6685 

97.2 

Y 

27 

5995 

97 . 3 

Y 

28 

9  1  50 

97.4 

Y 

29 

6750 

97  .  4 

Y 

30 

6685 

98.0 

Y 

31 

6105 

98.2 

Y 

32 

5970 

98 . 3 

Y 

33 

6685 

98.4 

Y 

34 

5977 

98.4 

Y 

35 

6230 

98  .  5 

Y 

36 

5975 

98.6 

Y 

37 

4140 

98.7 

Y 

38 

5995 

98  .  9 

Y 

39 

9150 

98 . 9 

N 

40 

5995 

99.0 

.  Y. 

41 

6230 

99.0 

42 

597  5 

99  •  1 

43 

5995 

99  .  1 

r 

T 

=  4+8+19+39 

=  70 

X 

m 

=  4 

N 

=  35 

n 

=  81 

Z 

=  -2.117 

44 

n 

m 

q  a  i 

99  :  2 

V 

y 

3439 

9  9.4 

V 

47 

3439 

99  .  4 

Y 

48 

6685 

99  .  5 

Y 

49 

6680 

99  .  5 

Y 

50 

6240 

9  9.9 

Y 

5  1 

67  50 

99 . 9 

Y 

52 

59  70 

100.0 

Y 

53 

6130 

100.0 

Y 

54 

6680 

1-00 .0 

Y 

55 

5940 

100.0 

Y 

56 

6240 

100.0 

Y 

57 

6150 

100.0 

Y 

58 

6230 

100.0 

Y 

59 

5975 

100.0 

Y 

60 

6210 

100. c 

Y 

61 

5975 

100.0 

Y 

62 

6230 

100.0 

Y 

63 

4140 

100.0 

Y 

64 

6695 

100 .0 

Y 

65 

6680 

100.0 

Y 

66 

6105 

100.0 

Y 

67 

5995 

100.0 

Y 

68 

5995 

100.0 

Y 

69 

7310 

100.0 

Y 

70 

3610 

100.0 

Y 

71 

6680 

100.0 

Y 

72 

3444 

100.0 

Y 

7  3 

7690 

100.0 

Y 

74 

9150 

100.0 

Y 

75 

6105 

100.0 

Y 

76 

5975 

100.0 

Y 

77 

9150 

100.0 

Y 

78 

6685 

100 .0 

Y 

79 

9330 

100.0 

Y 

80 

6685 

100.0 

Y 

81 

2040 

100.0 

Y 

82 

66  10 

100.0 

Y 

83 

6635 

100.0 

Y 

84 

1560 

100.0 

Y 

85 

6220 

100. o 

Y 
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TABLE  C-3.  Test  Data  for  Delivery  Ratings  atDGSC 


dank  case 


6 

7 

a 

q 

10 
1 1 
12 
13 
M 

15 

16 
17 
ia 

19 

20 
21 
22 
23 
2« 

25 

26 
27 
2B 

29 

30 

31 

32 

33 

34 

35 

36 

37 

38 

39 

40 

41 

42 

43 

4  4 

45 

46 

4/ 

4e 

49 

50 
:  I 


AV9 

OURS 

IU6H 

*  ins 

0  A  Y  S 

SCORE 

PERFORMED 

54.3 

82.0 

40 . 1 

N 

53.6 

77.0 

41.4 

H 

52.9 

78  .  « 

46.9 

fl 

54.8 

62 . 5 

47.9 

n 

30.3 

122.7 

48.2 

N 

61.5 

66.0 

50.5. 

N 

44.9 

41.0 

50.5’ 

N 

52.5 

50.6 

'  51.3 

Y 

39.1 

29.3 

51.7 

N 

33.6 

20.2 

i  52.1 

Y 

90.2 

107. 1 

54.1 

M 

25.3 

17.0 

54 . 4 

Y 

92.7 

251.0 

55.6 

N 

92.7 

110.0 

55.6 

N 

77.4 

76.3 

55.9 

H 

56.4 

47.5 

56 . 0 

N 

73.8 

66.3 

57.8 

N 

73.8 

65.7 

**  58.0 

N 

69.2 

55.9 

59.2 

N 

80.2 

65.6 

61.9 

H 

69.4 

46.6 

63.0  . 

n 

55.7 

25.7 

63.1 

Y 

96.2 

86.0 

63.3 

N 

56.0 

25.2 

63.5 

T 

75 .7 

54.0 

63.8 

Y 

57.7 

24.8 

64.7 

Y 

70.6 

41.1 

65.9 

Y 

69.2 

39.0 

65.9 

Y 

69.7 

39.2 

66. 1 

N 

60.7 

24.1 

66.8 

Y 

68.9 

34.0 

67.7 

Y 

63.5 

25.8 

67.8 

Y 

70.1 

34.? 

68.  1 

Y 

61.6 

20.5 

68.8 

Y 

61.5 

20.2 

68.8 

T 

63.8 

22.9 

69.1 

T 

88.0 

59.1 

69.2 

H 

72.8 

36.3 

69.2 

Y 

75.1 

39.2 

69.4 

N 

82.  1 

48 .  o 

70.2 

Y 

79.8 

44.0 

70.3 

N 

60.0 

I2.« 

70.6 

Y 

78.4 

«0.2 

71.0 

H 

78.5 

39.7 

7  1.2 

N 

61.9 

14.6 

71.2 

r  . 

7e.s 

39 . 0 

7  1.6 

Y-' 

74.3 

32.4 

71.9 

Y 

54.5 

16.2 

72.2 

r 

79.3 

27  .  1 

.'2.4 

N 

*  "  • 

29.3 

7:  .  5 

V 

?  2  .  3 

59  .  ; 

'2.6 

N 

7  2  J 

24.3 

*>  ** 

M 

3A«r 


5? 

54 

55 

56 

57 
5B 

59 

60 
61 
62 

63 

64 

65 

66 
67 
6B 

69 

70 

71 

72 

73 

74 

75 
7  6 

77 

78 

79 

80 
SI 
82 

83 

84 

85 

86 
87 

98 

89 

90 

91 

92 

93 

94 

95 

96 

97 

98 

99 
too 

tOI 
102 
'.0  2 

104 

105 

106 

107 

108 


CASE 


SOM- 

A  V  S 

”  I  ME 

Days 

2'.  .  0 

51.5 

- 

5C  .  ' 

2  .  3 

7  5.9 

29.6 

* : .  1 

23.0 

59 . 7 

19.2 

*5.0 

29.6 

86.3 

4  1.7 

66.3 

11.7 

76.8 

27.2 

97.9 

42 . 0 

94.0 

36.6 

30.4 

30.9 

83.0 

33.8 

69.6 

13.7 

78.1 

25.6 

84.  | 

34.2 

94.6 

49  .  1 

90.8 

28 . 4 

75.3 

20.2 

72.1 

14.8 

79.5 

25.4 

78.7 

24.3 

79.6 

25.1 

80.7 

26.2 

78.1 

22.5 

82.0 

2B  .  1 

76.3 

19.1 

78.8 

22 . 5 

74.5 

15.0 

91  .  9 

25.5 

90.5 

23.3 

96.0 

30.9 

72.5 

10.8 

92.4 

40. 3 

90.6 

37.6 

93.0 

26.2 

76.5 

16.5 

79.8 

20.9 

87.5 

32.0 

75.9 

13.  7 

79.0 

16.  B 

79.9 

17.9 

94.  4 

25.8 

74.0 

10.0 

77.4 

14.2 

31.0 

18.8 

34  .  2 

22.9 

?!.« 

23.0 

79.3 

:  1 

;  4  .  2 

u  - 

22  .  ? 

5*  .  a 

21.0 

97.5 

c  l  .  ? 

90.2 

i«.3 

92.7 

19.1 

OURS  HICH 

SCORE  PERFORr 

*3.5  •; 


”.6 
—  .  i 

74.1 

'4.| 

74.1  Y 

r*.2  T 

75.1  y 

75.1  Y 

75.2.  T 

*5-5"  r 

75.8  n 

75.9  T 

76.3  N 

76.3  T 

76.6  T 

76.8  r 

77.1  Y 

77.1  Y 

77.1  Y 

77.3  N 

77.5  Y 

77.5  Y 

77.7  * 

77.9  Y 

77.9  Y 

78.0  Y 

78.1  Y 

78.3  Y 

78.7  Y 

78.9  Y 

79.0  Y 

'9.2  Y 

79.2  r 

79.3  Y 

79.3  Y 

79.3  Y 

79.3  Y 

79.5  Y 

79.7  N 

80.0  Y 

80.1  r 

80.2  Y 

30.3  r 

90.4  r 

80.8  r 

91  .  I  Y 

3  1.J  K 

?  !  .  5  r 

9  1.3  r 

32.1  M 

3  2-2  • 

3  2.2  1 

92.7  r 

32.3  y 

32.4  ? 
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TABLE  C-3.  (continued) 


RANK  CAS; 


iO.'i- 
T  ITS 


A  VS 
OATS 


0 VRS  Hl6H 

-CORE 


109 

04 

6 

20 

6 

82 

' 

1  10 

90 

- 

29 

6 

82 

6 

* 

1  1  1 

SO 

2 

1  3 

5 

82 

7 

f 

112 

93 

a 

1  8 

6 

82 

3 

r 

113 

88 

4 

25 

4 

82 

9 

r 

1  1  4 

90 

9 

27 

0 

83 

/ 

T 

115 

94 

3 

1  7 

3 

82 

7 

r 

116 

86 

1 

19 

6 

63 

8 

r 

I  17 

85 

5 

1  8 

9 

83 

8 

r 

118 

se 

3 

23 

0 

82 

9 

119 

a  • 

- 

21 

/ 

83 

9" 

Y 

120 

88 

4 

2  1 

5 

84 

4 

121 

85 

7 

1  7 

4 

84 

5 

Y 

122 

83 

6 

1  4 

2 

•  84 

5 

Y 

123 

82 

5 

12 

6 

84 

5 

Y 

124 

96 

3 

33 

0 

.84 

6 

r 

125 

89 

6 

22 

9 

'84 

6 

N 

126 

95 

1 

31 

0 

84 

7 

Y 

127 

83 

8 

12 

8 

85 

2 

Y 

128 

88 

5 

i  a 

6 

85 

7 

Y 

129 

85 

4 

1  3 

8 

85 

7 

N 

130 

85 

0 

12 

6 

86 

0 

» 

131 

89 

2 

16 

5 

86 

9 

T 

1  32 

91 

7 

20 

0 

.  •  87 

0 

Y 

133 

86 

6 

12 

0 

87 

2 

r 

l  34 

85 

9 

10 

5 

87 

3 

Y 

135 

92 

4 

18 

8 

87 

9 

Y 

136 

96 

7 

23 

6 

88 

6  ‘ 

Y 

137 

99 

7 

12 

8 

88 

7 

r 

138 

87 

5 

a 

1 

89 

3 

Y 

139 

92 

2 

14 

0 

89 

7 

rj 

140 

89 

2 

a 

7 

90 

0 

r 

141 

97 

6 

21 

0 

90 

2 

Y 

142 

98 

7 

21 

0 

90 

8 

Y 

143 

90 

4 

7 

6 

91 

2 

Y 

1  44 

30 

1 

6 

3 

91 

5 

Y 

145 

90 

0 

6 

0 

91 

6 

Y 

146 

98 

1 

18 

0 

91 

7 

Y 

147 

94 

1 

10 

3 

92 

1 

Y 

148 

94 

6 

9 

4 

93 

0 

r 

149 

94 

1 

a 

6 

93 

0 

Y 

150 

96 

2 

8 

a 

94 

2 

Y 

151 

99 

0 

8 

7 

95 

9 

Y 

152 

98 

9 

2 

0 

98 

5 

Y 

153 

98 

4 

0 

0 

99 

0 

Y 

1  54 

1  00 

0 

0 

0 

100 

0 

Y 

T  =  1  +  2+  .  .  .  7-1-9+11  +  1  3+1 4+ .  .  .  21  +  23  +  29  +  37  +  39  +  41  +  43  +  44  +  49  +  51  +  52  +  53  +  64  +  66  + 
X  73+92+100+103+118+125+129+139  =  1671 

m  =39  N  =  154 

n  =115  Z  =  -5.615 
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